Automatic Analysis and Classification of Obfuscated Bot Binaries

Botnets is a serious threat to Internet security. Popular defense strategies such as traffic filtering and malware detection all require a good understanding of the constituent bot binaries for creating the corresponding filter rules or signatures. This means that an effective analysis and classification process for bot binaries is needed for dealing with the threat of botnets. Unfortunately, the rampant usage of binary obfuscation these days has made the analysis and classification rather difficult. A simple string pattern matching or disassembly of the binary no longer suffices as the exact instruction sequence can be easily altered by obfuscation. In this work, we propose a new framework for automatic analysis and classification of bot binaries. The framework analyzes a bot binary’s runtime system call trace and uses the longest common subsequences between system call traces for the classification of bot binaries. The framework can effectively deal with obfuscated bot binaries. Experiment result shows that the framework can attain an overall 94% true positive rate and 93% true negative rate.